Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get subset of building data, not all. #334

Merged
merged 6 commits into from
Oct 24, 2024
Merged

Conversation

jamesturner246
Copy link
Contributor

@jamesturner246 jamesturner246 commented Oct 23, 2024

Description

This PR adds code for interfacing with Overture for building data, allowing the download of only a subset of data (constrained by a given bounding box), instead of the entire dataset for the country. The downloaded building.geoparquet file is significantly smaller as a result.

I left the original national downloader in for not, in case it's still useful.

I have not tested this yet, beyond peering inside the geoparaquet file (I run Linux, so can't run the GUI). Please could you confirm?

Some things to note:

  • There are two endpoints: S3 or Azure (I arbitrarily chose S3)
  • S3 region is fixed, as only us-west-2 is available
  • I haven't yet figured out the version parameter, how to auto pick latest, or if that's even possible without hacking
  • I'm not so familiar with the data formats, so extra optimisation is perhaps possible.

Fixes #133

Type of change

  • Documentation (non-breaking change that adds or improves the documentation)
  • New feature (non-breaking change which adds functionality)
  • Optimization (non-breaking, back-end change that speeds up the code)
  • Bug fix (non-breaking change which fixes an issue)
  • Breaking change (whatever its nature)

Key checklist

  • All tests pass (eg. python -m pytest)
  • The documentation builds and looks OK (eg. python -m sphinx -b html docs docs/build)
  • Pre-commit hooks run successfully (eg. pre-commit run --all-files)

Further checks

  • Code is commented, particularly in hard-to-understand areas
  • Tests added or an issue has been opened to tackle that in the future. (Indicate issue here: # (issue))

@codecov-commenter
Copy link

codecov-commenter commented Oct 23, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 86.91%. Comparing base (6346642) to head (6ab0307).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #334      +/-   ##
==========================================
+ Coverage   86.76%   86.91%   +0.15%     
==========================================
  Files          22       22              
  Lines        2199     2217      +18     
  Branches      280      280              
==========================================
+ Hits         1908     1927      +19     
  Misses        214      214              
+ Partials       77       76       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Collaborator

@barneydobson barneydobson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a -m downloads test (which passes) just to make sure.

LGTM! Thanks so much

@barneydobson barneydobson merged commit b5779c1 into main Oct 24, 2024
5 checks passed
@barneydobson barneydobson deleted the overture_buildings branch October 24, 2024 10:13
@jamesturner246
Copy link
Contributor Author

You're welcome. Glad it helped!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

New building downloader
3 participants